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Many problems in computing, service, and manufacturing sys¬ 
tems can be modeled via infinite repeating Markov chains with an 
infinite number of levels and a finite number of phases. Many such 
chains are quasi-birth-death processes (QBDs) with transitions that 
are skip-free in level, in that one can only transition between consecu¬ 
tive levels, and unidirectional in phase, in that one can only transition 
from lower-numbered phases to higher-numbered phases. We present 
a procedure, which we call Clearing Analysis on Phases (CAP), for 
determining the limiting probabilities of such Markov chains exactly. 

The CAP method yields the limiting probability of each state in the 
repeating portion of the chain as a linear combination of scalar bases 
raised to a power corresponding to the level of the state. The weights 
in these linear combinations can be determined by solving a finite 
system of linear equations. 


1. Introduction. This paper studies the stationary distribution of Class 
M Markov chains, which are continuous time Markov chains (CTMCs) * 1 hav¬ 
ing the following properties (see Fig. 1 and Fig. 2): 

• The Markov chain has a state space, £, that can be decomposed as 
£ = 1Z U AT, where 1Z represents the infinite repeating portion of the 
chain, and Af represents the finite nonrepeating portion of the chain. 2 
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1 The methodology presented in this paper can easily be modified to apply to discrete 
time Markov chains. 

2 We note that this partition is not unique. 
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• The repeating portion is given by 

TZ = {( m,j) : 0 < m < M, j > j 0 } 

where both M and jo are finite nonnegative integers. We refer to a 
state ( m,j ) € TZ as currently being in phase m and level j. For each 
j > jo, level j is given by 

Lj = {(0, j), (1, j),... , (M, j)}. 

Throughout this paper, we index phases by i, k , m , and u, and we 
index levels by j and £. 

• Transitions between a pair of states in Af may exist with any rate. 

• Transitions from states in Af to states in TZ may only go into states in 
Lj 0 , but may exist with any rate. 

• Transitions from states in TZ to states in Af may only come from states 
in Lj 0 , but may exist with any rate. 

• Transitions between two states in TZ that are both in the same phase, 
rn. (e.g., the “horizontal” transitions in Fig. 1 and Fig. 2) are described 
as follows, with q(x,y ) denoting the transition rate from state x to 
state y: 

X m = q((m,j),(m,j+ 1)) (0 <m<M, j > j 0 ) 

Hm = - 1)) (0 < m < Af, j>jo + l). 

• We express transition rates between two states in TZ , which transition 

out of a state in phase m to a state in another phase (e.g., the “vertical” 
transitions in Fig. 1 and the “vertical” and “diagonal” transitions in 
Fig. 2) using the notation ct m (Ai; A 2 ), where Ai > 1 is the increase in 
phase from mtora + Ai (he., the “vertical” shift) and A 2 € {—1, 0,1} 
is the change in level, if any, from j to j + A 2 (he., the “horizontal” 
shift). Note that Ai > 1 indicates that only transitions to higher- 
numbered phases are allowed, while A 2 £ {—1,0,1} indicates that 
each transition may change the level by at most 1 in either direction. 
More specifically, these transitions are described as follows: 

otm(i ~ m\ —1) = q((m,j), (i,j — 1)) (0 <m<i<M, j > jo + 1) 

a m (i-m-0) = q((m,j),(i,j)) (0 <m<i<M, j > jo) 

a m (i — m; 1) = q((m, j), (i, j + 1)) (0 <m<i<M, j > j 0 ). 

We will also use the shorthand notation 

M 

a m = ( a m(i ~ m] -1) + a m (i - m; 0) + a m (i - m\ 1)) 

i=m -\-1 
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throughout the paper to represent the total outgoing transition rate to 
other phases from states in phase m with level j > jo + 1. 

• The Markov chain must be ergodic. 

Markov chains in class M are examples of quasi-birth-death processes 
(QBDs), with increments and decrements in level corresponding to “births” 
and “deaths,” respectively. We say that transitions in class M chains are 
skip-free in level, in that the chain does not allow for the level to increase or 
decrease by more than 1 in a single transition. We also say that transitions 
in class M chains are unidirectional in phase, in that transitions may only 
be made to states having either the same phase or a higher phase in the 
repeating portion. Note however that phases may be skipped: for example, 
transitions from a state in phase 2 to a state in phase 5 may exist with 
nonzero rate. 

Many common queueing systems arising in computing, service, and man¬ 
ufacturing systems can be modeled with CTMCs from class M. For such 
systems, one often needs to track both the number of jobs in the system and 
the state of the server(s), where each server may be in one of several states, 
e.g., working, fatigued, on vacation, etc. When modeling a system with a 
class M Markov chain, we often use the level, j, of a state ( m,j ) to track the 
number of jobs in the system, and we use the phase, m, to track the state 
of the server(s) and/or the arrival process. For example, a change in phase 
could correspond to (i) a policy modification that results in admitting more 
customers, as captured by an increase in “arrival rate” from \ m to A j, where 
A i > \ m or (ii) a change in the state of the servers leading to an increase or 
decrease in the service rate from p m to //j. A few examples of systems that 
can be modeled by Class M Markov chains are presented in Section 2. 

1.1. The matrix-geometric approach. One way of studying the stationary 
distribution, n, of a class M Markov chain is to observe that it exhibits a 
matrix-geometric structure on 1Z. More specifically, if we let ttj represent the 
limiting probability of the states in Lj, that is, tv j = (7T(o,j), ^(ij), • • • , 7T( mj )), 
then for j > jo 


TV j —|— ] — 7T j 11 


where R <E R( M+1 )x( M +i) is referred to as the rate matrix associated with 
the chain. If we let the sojourn rate of state x be defined by 



y^x 


phase, 
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Fig 1. The structure of class M Markov chains. In this case jo = 0 and, for simplicity, 
a m (i — m; ±1) = 0. The chain is made up of a non-repeating portion, JV (shown here as an 
aggregation of states), and a repeating portion, TZ. Within 1Z, each phase, m, corresponds 
to a “row” of states, and each level, j, corresponds to a “column” of states. Transitions 
between levels in each phase of the repeating portion, TZ, are skip-free: all such transitions 
move only one step to the “left” or “right.” Transitions between phases in each level of TZ 
are unidirectional; all such transitions move “downward. ” The thicker arrows denote sets 
of transitions (transitions rates for these sets are omitted from the figure). 
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Fig 2. Another more detailed look at the transition structure of class M Markov chains. For 
simplicity, only the set of transitions that are possible from state ( m,j ) (where j > jo + 1) 
to states in phases m, m + 1, and m + 2 are shown. Note that all transitions from ( m,j ) 
are either to the left, to the right, or downward. Furthermore, all transitions can decrease 
or increase the level by at most one. 
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then we can describe the elements of R probabilistically as follows: the ele¬ 
ment, Ri,rm in row i, column m of R can be interpreted as V(i.j) times the 
expected cumulative amount of time the chain spends in state (■ m,j + 1) 
before making a transition into a level strictly below j + 1, given the chain 
starts in state (i,j)- For most QBDs, one cannot derive an exact expression 
for each element of R, but there are many ways to compute an approx¬ 
imation of R numerically: see for example [16, 4]. Readers interested in 
further details should consult the matrix-analytic texts of Neuts [20], La- 
touche and Ramaswami [17], and He [13]. Queueing textbooks of a broader 
scope that also discuss matrix-analytic methods include Asmussen [3] and 
Harchol-Balter [12], Once R —or good approximations for R —have been 
found, then ttj = 7fj 0 R J_ ' 50 for j > jo, and so all remaining limiting prob¬ 
abilities, ir x , for x £ JV, can be found using the balance equations and the 
normalization constraint. 

There are many examples of QBDs with a rate matrix, R, that can be 
computed exactly through a finite number of operations. One class of QBDs 
having a closed-form rate matrix is presented in Ramaswami and Latouche 
[22], with an extension to Markov chains of GI/M/ 1-type given in Liu and 
Zhao [19] . Other classes of QBDs having explicitly computable rate matrices 
are considered in the work of van Leeuwaarden and Winands [30] and van 
Leeuwaarden et al. [29], with both of these studies being much closer to 
our work, since most (but not all) of the types of Markov chains studied in 
[30], and all of the chains discussed in [29] belong to class M. In [30, 29] 
combinatorial techniques are used to derive expressions for each element of 
R that can be computed exactly after a finite number of operations, but 
their methods are not directly applicable to all class M Markov chains as 
they further assume that X m and p m are the same for 0 < m < M — 1, 
and they also assume that for each 0 < nn < M — 1, any transitions leaving 
phase m must next enter phase m +1 (i.e., they assume phase transitions are 
skip-free —in addition to being unidirectional—within the repeating portion 
of the chain). 

Even closer to our work is the work of Van Houdt and van Leeuwaarden 
[28], which presents an approach for the explicit calculation of the rate ma¬ 
trix for a broad class of QBDs including those in class M. This approach 
involves solving higher order (scalar) polynomial equations, the solutions to 
which are expressed as infinite sums, which typically cannot be computed in 
closed-form. However, [28] also gives an approach for calculating closed-form 
rate matrices for a class of Markov chains called tree-like QBDs. Tree-like 
QBDs neither contain nor are contained by class M, although there is signifi¬ 
cant overlap between the two. Transitions between phases (within a level) in 
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tree-like QBDs form a directed tree, while transitions between phases in class 
M Markov chains form a directed acyclic graph. Specifically, unlike class M 
chains, tree-like QBDs do not allow for a pair of phases i ^ k to both have 
transitions to the same phase m . (i.e., tree-like QBDs do not allow for both 
ai(m — i; A) >0 and atk{m — k: A') > 0 when i / k and A, A' € {—1,0,1}). 

1 . 2 . Our approach: Clearing Analysis on Phases (CAP). In this study 
we introduce the CAP (Clearing Analysis of Phases) method for evaluating 
the stationary distribution of class M Markov chains. This method proceeds 
iteratively among the phases, by first expressing all 7T( 0 j) probabilities, for 
j > jo, in terms of n x probabilities for x G N. Once each element 7 :t m ,j) f° r 
a fixed phase m, j > jo has been expressed in terms of { 7 r x } xe jy, we then 
do the same for all 7r( m+lj -) terms. After each tt(mj) expression has been 
determined, we use the balance equations and normalization constraint to 
solve for the remaining {'n x } x &N' probabilities. CAP takes its name from the 
fact that, between two phase transitions, class M Markov chains behave like 
an M/M/1 /clearing model, that is, each phase is likened to a birth-death 
process that experiences “clearing” or catastrophic events in accordance to 
an independent Poisson process. In our model, these “clearings” corresponds 
to a change in phase. 

One major advantage of the CAP method is that it avoids the task of 
finding the complete rate matrix, R, entirely, while yielding expressions 
for 7 T( m ,j) that only involve raising M + 1 scalars to higher powers. There 
exists one such scalar, r m , for each phase, m € {0,1,..., M}. These scalars, 
referred to throughout as base terms, are actually the diagonal elements of 
the rate matrix, R, i.e., 

r m = Rm,mi (0 < m A Af) 

and the transition structure of class M Markov chains makes these elements 
much easier to compute than any of the other nonzero elements of R. Fur¬ 
thermore, the structure of 7 Tr m j\ depends entirely on the number of base 
terms that agree with one another. For example, when all nonzero base 
terms are distinct, one can show that 

m 

(l- 1 ) 7T {m ,j) = £c m , fc rr°, 

k =0 

for 0 < m < M, where the {c m) k}o<k<m<M values are constants that do 
not vary with j, and can be computed exactly by solving a linear system of 
0(M 2 + \Af\) linear equations. 
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In the case where all base terms agree, we instead find that 


( 1 . 2 ) 





3 ~ Uo + 1 ) + k\ 
k ) 


7 0 ) 


where again, the c m ^ terms can be computed by solving a linear system. 

In retrospect, it is of no surprise that can be expressed as a linear 

combination of scalars, each raised to the power of j — jo, as in Equations 
(1.1) and (1.2): R must be upper-triangular for class M chains. This follows 
by observing that R t , m is i times the expected cumulative amount of 
time spent in state (m,j o + 1) before returning to Lj 0 , given initial state 
(i, jo), and this value is 0 when i > m. Since R is upper-triangular, its 
eigenvalues are simply its diagonal elements, which are also the diagonal 
elements of the Jordan normal form of R—see e.g., Chapter 3 of Horn and 
Johnson [14]— from which we know that 7 T( m ,j) can be expressed as a lin¬ 
ear combination of scalars, each raised to the power of j — jo- Although in 
theory, our solution form could be recovered by first computing R and then 
numerically determining R in Jordan normal form, such a procedure is often 
inadvisable. The structure of the Jordan normal form of a matrix can be 
extremely sensitive to small changes in one or more of its elements, partic¬ 
ularly when some of its eigenvalues have algebraic multiplicity larger than 
one, as is the case for all of the models discussed in [30, 29]. Fortunately, the 
CAP method can handle these cases as well with little additional difficulty. 

The statement and proofs of this paper’s main results are presented in 
Section 3. This proof relies on some results regarding M/M/1 /clearing mod¬ 
els; the proofs of these results are deferred to Section 4. In Section 5 we 
briefly touch upon how the CAP method may be applied to chains beyond 
those in class M. 


1.3. Recursive Renewal Reward, ETAQA, and other techniques. We brief¬ 
ly review existing techniques for solving QBDs beyond the matrix-geometric 
approach and comment on their connection to the CAP method. 

Gandhi et al. [9, 10] use renewal theory to determine exact mean values 
and z-transfornrs of various metrics for a subclass of chains in M via the 
Recursive Renewal Reward (RRR) method. The class of chains they study 
do not allow for “diagonal” transitions (i.e., a m (i — m;±l) = 0). Unlike 
our method, RRR cannot be used to determine a formula for a chain’s 
limiting probability distribution in finitely many operations. While there 
is overlapping intuition and flavor between CAP and RRR—both methods 
make use of renewal reward theory—CAP is not an extension of RRR and 
does not rely on any of the results from [9, 10]. 
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The Efficient Technique for the Analysis of QBD-processes by Aggrega¬ 
tion (ETAQA), first proposed by Ciardo and Simirni [7], combines ideas 
from matrix analytic and state aggregation approaches in order to compute 
various exact values (e.g., mean queue length) for a wide class of Markov 
chains. By design, ETAQA yields the limiting probability of the states in 
the non-repeating portion, AT, along with the limiting probabilities of the 
states in the first level (or first few levels) of the repeating portion, 1Z. The 
limiting probabilities of the remaining states (i.e., higher level states) are 
aggregated, which allows for the speedy computation of exact mean values 
and higher moments of various metrics of interest. In particular, ETAQA 
involves solving a system of only 0(\J\f\+M) linear equations. Although orig¬ 
inally applicable to a narrow class of chains (see [7, 6] for details), ETAQA 
can be generalized so as to be applicable to M/G/l-type, GI/M/1-type, and 
QBD Markov chains, including those in class M (see the work of Riska and 
Smirni [23, 24]). Stathopoulos et al. [27] show that ETAQA is also well suited 
for numerical computations; ETAQA can be adapted to avoid the numer¬ 
ical problems alluded to in Section 1.1. Unlike the CAP method, ETAQA 
(like RRR) cannot be used to determine a formula for a chain’s limiting 
probability distribution (across all states) in finitely many operations. 

For certain class M Markov chains, one can also manipulate generating 
functions to derive limiting probabilities, such as in the work of Levy and 
Yechiali [18] and the work of Phung-Duc [21], where this type of approach 
is used to solve multi-server vacation and setup models, respectively. This 
approach is covered in greater generality in a technical report by Adan and 
Resing [2]. We note that although generating function approaches can yield 
solutions of a form similar to those found using the CAP method, the two 
approaches differ in methodology. 

2. Examples of class M Markov chains. In this section we provide 
several examples of queueing systems which can be modeled by class M 
Markov chains. In each example we will use the phase, m G {0,1,..., M}, 
to track the “state” of the server(s) and/or the arrival process, and the 
level, j, to track the number of jobs in the system. Of course, there are 
many systems beyond those covered in this section that can be modeled by 
class M Markov chains. For example, class M chains were recently used to 
model medical service systems in [8], [5], and [25]. 

2.1. Single server in different power states. Consider a computer server 
that can be in one of three different power states: on, off, or sleep. In the 
on state, the server is fully powered and jobs are processed at rate /i. 
In the off state, the server consumes no power, but jobs cannot be pro- 
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number of jobs, j 



Fig 3. The Markov chain for a single server in different power states. State (m, j ) indicates 
that the server is in state m (0=off, l=sleep, 2=on) with j jobs in the system. 


cessed. When the server is idle, it is desirable to switch to the off state in 
order to conserve power, however there is a long setup time, distributed 
Exponential^), needed to turn the server back on when work arrives. Be¬ 
cause of this setup time, it is common to switch to a state called the sleep 
state, where the server consumes less power than the on state, but where 
there is a shorter setup time, distributed Exponential(e)), for turning the 
server on. It is also common to purposefully impose a waiting period, dis¬ 
tributed Exponential(/3), in powering down a server (from on to sleep, and 
again from sleep to off) once it is idle, which is useful just in case new jobs 
arrive soon after the server becomes idle. See [11] for more details. 

Fig. 3 shows a Markov chain representing this setting. This is a class M 
chain with M + 1 = 3 phases: off (m = 0), sleep (m = 1 ), and on (rn = 2). 
For this chain, jo = 1 and the non-repeating portion of the state space is 
N = {(0,0), (1,0), (2,0)}, while A 0 = Ai = A 2 = A, fi 0 = m = 0 , /j 2 = ft, 
a 0 (2- 0) = 7 , and ou(l; 0) = 5 > 7 (all other a m (m — i\ A) transition rates 
are zero). 

The system becomes much more interesting when there are multiple servers, 
where each can be in one of the above 3 states. In the case of 2 servers, there 
will be 6 phases, corresponding to: (off,off), (off,sleep), (off,on), (sleep,sleep), 
(sleep,on), (on,on). Note than in this case, phase transitions will include 
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number of customers, j 



Fig 4. The Markov chain for a server susceptible to fatigue. State ( m,j ) indicates server 
state m (0=full speed, l=reduced speed, 2=slow speed) with j customers in the system. 


transitions with rates 2 y, 7 + 5, and 26, as both servers may be attempting 
to turn on at the same time. In general, a system with a servers and b server 
states will have ( a+ ( ^ -1 ) phases. 

2 . 2 . Server fatigue. Consider a human server who starts her shift full of 
energy and works quickly (at rate rf)- As time passes and fatigue sets in, 
she gets slower and slower (first she slows down to a reduced rate hr and 
eventually to a very slow rate rs, where rs < TR < Tf)- At some point 
it makes sense to replace her with a fresh human server. However, before 
we can do that, she needs to finish serving her queue of existing customers, 
while no longer accepting further arrivals. We assume that the time it takes 
for the new replacement to start working is distributed Exponential(/3). 

Fig. 4 shows a Markov chain representing this setting. This is a class M 
chain with M + 1 = 3 phases: full speed ( m = 0), reduced speed (m = 1), 
and slow speed (m = 2). For this chain, jo = 1 and the non-repeating 
portion of the state space is M = {(0,0), (1,0), (2,0)}, while Ao = Ai = A, 
A 2 = 0, ho = fJ-F, Hi = TR < TF, H2 = HS < TR, «o(l:0) = 7 and 
«i(l;0) = 6 (all other a m (m — i; A) transition rates are zero). 

Again, the system becomes much more interesting when there are multiple 
servers, where each can be in one of the above 3 states. 
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uninfected 


undetected 

infection 


detected 

infection 


number of jobs, j 



Fig 5. The Markov chain for a server vulnerable to viruses. State ( m,j ) indicates server 
state m (0=uninfected, l=undetected infection, 2=detected infection) with j jobs in the 
system. 


2.3. Server with virus infections. Imagine a computer server that is vul¬ 
nerable to viruses. We present a stylized model where normally, the server 
is uninfected and receives jobs with rate A and processes them with rate 
ft. While most jobs are normal (i.e., not virus carriers), arriving at rate 
A at, every once in a while, one of the arriving jobs brings with it a virus, 
with rate Ay = A — Ajv- The virus causes the server to become infected, 
reducing the server’s service rate from p to ft j. It takes a duration of time 
distributed Exponential^) for the server to detect that it is infected. Once 
the infection is detected, the server stops accepting new jobs, and once all 
remaining jobs are processed, the server is able to use antivirus software to 
remove the virus in a duration of time distributed Exponential^). Once the 
virus is removed, the server is again uninfected and will resume accepting 
jobs, processing them at a restored service rate of ft. We model a single 
server as being in one of 3 states, each of which will make up a phase of our 
Markov chain: uninfected (m = 0), undetected infection (m = 1), and 
detected infection (m = 2). 

Fig. 5 shows a class M Markov that represents this setting. For this chain, 
M = 2, jo = 1, A r= {(0,0), (1,0), (2,0)}, A 0 = X N , X 1 = X = X N + Ay, 
A 2 = 0, [iq = fi, fi\ = ji 2 = fti, cto(l; 1) = Ay, and ai(l;0) = 0 (all other 
a m (m — i] A) transition rates are zero). 
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3. Results. In this section we first present a key theorem from the 
literature that enables the CAP method (Theorem 1). We then introduce 
some preliminary notation, and an original result, Theorem 2. Finally, we 
present the main results of our paper, Theorems 3, 4, and 5, the proofs of 
which will depend on both Theorems 1 and 2. 


3.1. A key idea. Consider an ergodic CTMC with state space, S , and 
consider a nonempty proper subset, ACS, with states x, z € A. The CAP 
method involves calculating quantities of the form 

' cumulative time spent in state x until next" 
transition leaving set A, given initial state z_ 

in order to determine the limiting probabilities of the Markov chain of in¬ 
terest. Theorem 1 (from Theorem 5.5.1 of [17]) gives an expression for the 
limiting probabilities of the Markov chain in terms of the quantities E 2 [ T .. 

Theorem 1 . Suppose ACS. Then for each x C A, the limiting proba¬ 
bility of being in state x, n x , can be expressed as 

*x = Y Y z)^ z [T x ] , 

y£A c z€A 

where q(y,z) is the transition rate from state y to state z and A c = S\A. 
Proof. See Theorem 5.5.1 of [17]. □ 


E 2 [T^] = E 


Intuitively, we are expressing the long run fraction of time that we reside 
in state x, tt x , as a weighted average of the cumulative time spent in state x 
during uninterrupted visits to states in A, E 2 [T^], conditioned on the choice 
of state, z C A, by which we enter A. The weights in this average represent 
the rate at which visits to A via z occur, which involves conditioning on the 
states y C A c by which one may transition to z € A. We illustrate S, A, y , 
z, and x in Fig. 6. 

As an example, consider the simple case where A = {x}. In this case, 
Theorem 1 yields 


X 


Y Y n y q ( y ’ z ^ Ez [ t * a ] = Y x ) E * 

y£A c zGA 



J2 y ^ x Q(x,y) 


k x Y y) = Y n y q (yi x )’ 

yA x y^x 


and so 
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Fig 6. For any x £ A, Theorem 1 gives n x as a linear combination of quantities E z 
by conditioning on the states y £ A c , by which one may transition to states z £ A. This 
figure shows one such (y, z) pair. 

which is simply the balance equation associated with state x. 

Theorem 1 is used in [17] to establish the matrix-geometric structure 
of the stationary distribution of QBD chains. The same argument can be 
used to establish the matrix-geometric structure satisfied by the stationary 
distribution, n, of a class M Markov chain on 1Z. Fix a level j > jo, and 
define A = U^>j+i ^£- Then for each state (m, j + 1) G Lj +\, we have 


M M 



i =0 k =0 



M 



thus proving that Ttj+i = since R t , m , is vuj) times the expected amount 
of time the chain spends in state ( m,j + 1) before returning to Lj, given 
it starts in state (z,j), and R is the rate matrix whose (i,m)th element is 
given by R i)m . 

We will soon see that the CAP method consists of applying Theorem 1 
by choosing the set A in a different manner, while simultaneously observing 
that the resulting expected values of the form K z [T!^] can be reinterpreted 
as tractable expected values associated with an M/M/1/clearing model. 
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3.2. Preliminaries. Our main results, and their proofs, will rely on the 
following notation: 

• P m = {(to, jo + 1), (m, jo + 2), (to, jo + 3),...} is the set of states in 
phase m with level j > jo + 1 (i.e., the set of states in phase rn of 1Z 
excluding state (to, jo)). 

• Pm = X m j pm- 

• 0 m (-) is the Laplace Transform of the busy period (time to first reach 
state 0, given that one starts in state 1) of an M/M/1 Markov chain 
with arrival rate X m and departure rate p m : 


S + X m + p m — y/(s + X m + Pm ) 2 ~ 4A m Prn 

<Pm{S) = -XT-• 

The bases of our main theorem, r m , are given by 
(3.1) 


r m = 


) if p m > 0 

Am if = o, 


h/,; + ey. m 


recalling that 


M 


1 


a m = ^2 a rti(i ~ m; A). 

i=m +1 A= —1 


For convenience, we define the following quantity, which will appear 
frequently in our analysis: 


(3.2) 


n i — 


^m( 1 1~m ( t > m{o : m)) 


As a consequence of the ergodicity assumption on class M Markov chains, 
we have 


• for any phase to, X m > Pm implies a m > 0 , 

• and for any phase to, \ m = 0 implies that there exists a phase, i < to, 
and A G {—1,0,1} such that cq (to — i\ A) > 0. 

We also make the following observations: 

• r m < 1 for all phases, to G {0,1,..., M}. 

• i'm = 0 if and only if A m = 0. 

• i'm = Pm whenever a m = 0 (e.g., when to = M, as otM = 0). This is 
because a m = 0 implies that p m > Xm. by the ergodicity assumption, 
which yields </> m ( 0 ) = 1 . 
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• 4>m(s) = 0 for all s whenever X m > [i m = 0, which follows from the 
expression for Alternatively, this follows by observing that the 

busy period of a degenerate (non-ergodic) M/M/1 Markov chain with 
arrival rate X m = 0 is infinite. 

We have the following fundamental result on class M Markov chains, 
which together with Theorem 1, will enable us to prove the main results of 
our paper (Theorems 3, 4, and 5). 


Theorem 2. For any Markov class M Markov chain, if X m , Hm > 0 and 
£,j > jo + 1? we have 


(3.3) 




n^Pm 
1 {m,j) 


VL m rin 1 (l - {r m (j) m (a m )Y jo ) if £ < j 

kl m (t)m{oi rn ) l ~ j (1 - (r m (j) m (a m )) j ~ jo ) if i> j. 


with r m as given in (3.1) and as given in (3.2). 


Proof. The proof of this result is deferred to Section 4, which is entirely 
focused on proving this result via clearing model analysis. □ 


The remainder of this section will present our main results, giving the 
stationary distribution of class M chains via the CAP method in three 
different cases. In Section 3.3, we consider the case where all bases, r m , 
are distinct whenever they are nonzero. Distinct bases arise in many mod¬ 
els where there is no structure connecting the transition rates associated 
with each phase. For example, the class M Markov chain representing the 
“server in different power states” model presented in Section 2.1 has distinct 
bases. In Section 3.4 we consider the case where all bases are the same (i.e., 
A) = r\ = • • • = r m ), while requiring that X m , fj, m > 0, for simplicity. We 
study this setting because it is the simplest case featuring repeated nonzero 
bases. Finally, in Section 3.5 we proceed to the case where all bases except 
for vm are the same (i.e., tq = r\ = ■ ■ ■ = vm -i 7 ^ ?’m)- This structure, 
which is studied in [30, 29], is common in settings where phase transitions 
are analogous across all phases, except for the final phase where there are 
no transitions to a further phase before the process transitions to the non¬ 
repeating portion. In this case, we again assume that A m ,/r m > 0, for sim¬ 
plicity. While in principle, the CAP method can be used to determine the 
limiting probabilities of any class M Markov chain, for simplicity, we do not 
cover other cases (e.g., rq = rq / r^ = r$ = rj ^ r± = r& ^ r\), as the 
computations become increasingly cumbersome. 
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3.3. The case where all nonzero bases are distinct. We are now ready 
to present our main result for the case where all nonzero bases, r m , are 
distinct. Theorem 3 expresses the stationary distribution of such class M 
Markov chains as the solution to a finite system of linear equations. 

Theorem 3. For any class M Markov chain such that all nonzero bases 
r\,r 2 , ■ ■ ■ ,rM—given in Equation (3.1) — are distinct (i.e., r m ^ ri implies 
either m ^ i or r m = A m = 0), for all j > jo + 1, we have a limiting 
probability distribution of the form 


f Ki) 


E 

k =0 


'1 — 1 o 

r ryvj J 

m,k' k 5 


where {c my k}o<k<m<M are constants with respect to j. Moreover, together 
with jq ) }o<m<M and x}x£j\f> the {c m t k}o<k<m<M values constitute 

M(M + 5)/2 + \J\f\ + 2 “unknown variables” satisfying the following system 
of M(M + 5)/2 + \Af\ + 3 linear equations: 


C"m,k 


Cm,k 

^ m,k 0 


( m—1 1 

T. E - i\ A)rjf 

i=k A =—1 
Am ^m)( 1 
m — 1 1 

E E c i,kai{m - i-, A)r£ 

i=k A= —1 


Mm( 1 -r k ) + a n 


Cm,m — q) ^ ^ c m,k 


(0 < k < m < M : r m , r & > 0) 

(0 < k < m < M: Tk > = 0) 

(0 < k < m < M: = 0) 

(0 < m < M) 


k=o 


m m — 1 0 

Um JO Cm.kTk + E ?(*> jo))Kx + JQ JQ aj{m - V, A)w(i,j 0 _A) 

fc=° iSAT i=0 A=-l . . 

0) = -m -I- (0 < m < M) 

\m+ JO E a m(i — m; A) + JQ q((m,jo),x) 

i=m -\-1 A=0 xGAC 


E 4 l (( m >.7o)> x )’ r (m,j 0 ) + J)) q(y,x)n y 

m =0 j/GjV 

M 

E q(x,(m,j 0 )) + E q(x,y) 

m =0 i/GAT 


(s G JV) 


1 = e ^ + E E 

xGA/" m=0 k =0 


Cm,k 
1 - r k ’ 


where q(x,y) denotes the transition rate from state x to state y. 
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We note before proving Theorem 3 that solving this system of equations 
symbolically will yield closed-form solutions for the limiting probabilities. 
Alternatively, if all parameter values are fixed and known, an exact numer¬ 
ical solution can be found by solving the system numerically using exact 
methods. Note that there is one more equation than there are unknowns, 
as is often the case in representations of limiting equations through balance 
equations. Although one equation can be omitted from the system, the nor¬ 
malization equation must be used in order to guarantee a unique solution. 

It is also worth observing that once the values {^ x }xeM and {7T( m j 0 )}o <m<M 
are known, all other c m) k terms can be computed recursively, without having 
to apply Gaussian elimination to the entire linear system given in Theorem 
3. 

This recursion may also simplify further for some types of class M Markov 
chains. For example, if a m (Ai; A 2 ) = 0 for all Ai > 2, A 2 G {—1,0,1}, and 
0 < m < M , then when all bases are positive, for any k < m, we have 

1 

_ HTm /1 A \ A 

X m (n - r m )(l - L ,®”" 1 ’ K 

which further implies, for k < m, 

m—k 1 

n r k r k+e ST' A\ A 

t „ Wr* - A+d(l - ^ 

meaning that only the {ck : k}o<k<M terms need to be computed recursively. 


Proof of Theorem 3. For simplicity, we present the proof for the case 
where X m , n m > 0 for all phases m G {0,1, 2,... , M}. The complete proof 
that includes the cases where one or both of A m and /i m may be 0 for some 
phases, m, is given in Appendix B. 

We prove the theorem via strong induction on the phase, m. Specifi¬ 
cally, for each phase m, we will show that n( m ,j) takes the form 7 = 
c m,kr 3 k ~ 30 for all j > j 0 + 1, and show that {c m!fc } 0 <fc< m -i satisfies 

( m— 1 1 

V V Ci,kOti{m - i; A)rf 

■ 2^—_x 

while c m>m = 7To — Yl™=o °m,k■ Finally, after completing the inductive proof, 
we justify that the remaining linear equations in the proposed system are 
ordinary balance equations together with the normalization constraint. 
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Base case: 

We begin our strong induction by verifying that the claim holds for the 
base case (i.e., for m = 0). In this case, Equation (3.3) yields 


E (OJo+l) 


rnPo 

i (0j) 


Ho r J 0 jo X (1 - ro^o(ao)) = 


We can now apply Theorem 1, yielding 


7 r (0j) = 7 r (Ojo) A oE(Oj 0 +l) 


rpPo 

1 (pjy 


= Co,o^o Jc \ 


7r (0,io) A 0 



7 r ( 0 ^o) r O 


where co,o = ^(ojo) - Hence, tt ( 0 n takes the claimed form. Moreover, co,o 
satisfies the claimed constraint as co,o = 7r (o,j 0 ) — 'Pk=o c m,k = 7T(oj 0 ) — 0 = 
7 T(oj 0 ), because the sum is empty when m = 0. Note that when m = 0, 
{cm,fc }o<fc<m<M is empty, and hence, there are no constraints on these values 
that require verification. 


Helpful computations: 

Before proceeding to the inductive step, we compute two useful expres¬ 


sions: First, we have A m E( mj0+1 ) 




— r J~J0 

— 1 m 


, which follows from ap¬ 


plying Equation (3.3). Next, we have 


E 


r k 


(m ,j) 


= E 


r k J ° E {m ,e) 


Tf 


o+l 


E ; 

t=3+1 


JOICV 

k 


T, 




= E r k J0 ) 

\t=jo+l 


+ E r k 30 4>rn{.OLmY J (l “ (rmA»K)) J °°) 

*=j+l 
r-fer m (r 


fc m 


Am(^fc r m )(l ^ , m(o m )r/c) 


where the last equality follows from well known geometric sum identities. 
Note that this expression is well-defined because r k / r m by assumption 
and r m (j)m{oim) 7 ^ 1- 

Inductive step: 

Next, we proceed to the inductive step and assume the induction hypoth¬ 
esis holds for all phases i € {0,1,... , m — 1}. In particular, we assume that 
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= Ylk=o c i,k r l " , ° f° r i < m. Applying Theorem 1, the induction 
hypothesis, and our computations above, we have 3 


m— 1 oo 1 


7V (rn,j) — 7r (m,io)^mE(m ) j 0 + l) ^ ^ ^ 

i=0 1=1 A= —1 

m — 1 oo 1 / i 

'm 30 + J2J2 £ ai{m - i-, A) c i* r i~ 

i=0 £=1 A= —1 \k=0 


7 T/ • \ r*-^ 


£-j 0 -A 


E 


(m,£) 


[ t &A 


k =0 i=k 

m — 1 m — 1 


m —i m — l /l \ / oo 

^(mjo)^ 30 + J2 ( Ci A £ A)r£ [ T ^m,j)\ 


A= —1 
1 


7T( m,j 0 )r 3 m 30 + ^ ^ Cj.fc ^ Oj (m - v, A)r k 


rkr m (r J k 30 - r J m J0 ) 

A m{j'k ^77l)(l 0771 (^771 )^*fc) 


= 2_^ C m,k r i 30 , 


= £< 
fc =0 

where we have collected terms with 

/ m—l 1 

TkTm £ £ Ci tk Ui{m. - i; A)r 


Cm.k — 


, 2=/c A= —1 


(0 < k < m < M) 


^ m(j'k 7 m )(l ^ 771 ( 0771 ) 7 ^) 

and c m)m = 7 T( m j 0 ) — c m,fc) as claimed. This completes the inductive 

step and the proof by induction. 


The balance equations and normalization constraint: 

The equations with 7 r( mj0 ) and 7r x in their left-hand sides in our proposed 
system are ordinary balance equations (that have been normalized so that 
there are no coefficients on the left-hand side). 

It remains to verify that the final equation, which is the normalization 
constraint: 


M Moo 

1 = £ 7I " x + £ 7r (,m,j 0 ) + £ £ 7r (m,j) 

xdN m =0 m=0 j=jo+l 

MM M m—l oo 

= £ 7rx + ££ c v fc + £ £ £ 

x£j\f m=0 k=0 m=0 k=0 j=jo+l 

M TYl 

5>* + EEi 


„ , r J~J0 
L -m,k 1 k 


C-m,k^k 


x£A f m =0 k =0 


- T’fc 


□ 

3 Note that we have also used the fact that tt (i, 30 ) also satisfies the claimed form for all 
i < m, which is true as a,i = ^(i,j 0 ) — El=o Ci < k (fr° m the inductive hypothesis) implies 

that Tf(i : j 0 ) = Efe=o Ci , k = Efc=o c i.fc r fe- 
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3.4. The case where all bases agree. The CAP method can also be used 
in cases where some of the base terms coincide. We assume, for the sake 
of readability, that X m and n m are both positive for each phase m, but 
analogous results can still be derived when this is no longer the case. 

In order to derive our result, we will make use of the following lemma: we 
omit the proof, but each formula can be derived using the lemmas contained 
in Appendix C. 


Lemma 1. For a class M Markov chain with all A m ,// m > 0 and vq = 
r i = ■ ■ ■ = rM, for each integer u > 0 and each integer j > jo + 1, we have 
the following three identities: 


■ E 

£=jo+2 




u 


r 0 


r pPm 

1 ( m,j) 


u-\-l 

= E 


k =1 


XlmT 0 

(1 — rof>m(oim)) U+1 ~ k 


j - (jo + 1) + k\ j_ jQ 
k ) r ° ’ 


■ E 

e=h+i 

= a, 


— (jo + 1) + u \ l- 


u 


0 


'() 


j - (jo + 1) + u + 1 


J Jo 


+ E 


»+i >'» 

Sl m ro^m(a m ) (j - (jo + 1) + k 


—J (1 - ro<j) m (a m )) u+1 k 


k 


-JO 
r 0 ’ 


oo 

• E 

1=3 o+l 



(jo + l)+u\ ,_,o E 

u 1 


(m,£+ 1 ) 


T^Pm 

1 (m,j) 


^7TL fj~ (j0 + 1) + U + l\ ^j-j 0 _ /j — (j0 + 1) + U 

ro \ u +1 / 0 V u 

Xl m rQ<f> rn {pt iri ) f j (jo T 1) T k\ j—j 0 

“ (1 - r 0 (j) m (a m )) u+1 - k V k J 0 ' 


r j~jo 
M)_ 

Xm 


Theorem 4. For a class M Markov chain with all XFm > 0 and 
D = n = • • • = Lm? f or oil 0 < m < M, j > jo, we have 


n (rn,j ) 



j - (jo + 1) + 

k J 


M—jo 

r 0 
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where the {c mj k}o<k<m<M values satisfy the system of linear equations 


Cm, 0 — ^{m,j o) 


(0 < m < M) 


m— 1 m —1 


C-m,k E E 


'i—k i=u 
m —1 


^ ^ Qti(jTl i, (o; m ) 


.A 


(1 - ^O^m(o!m )) W+1 


-fc 


- t— 5Z a,kOn{m - i; 1) 


i=k 
m —1 


+ ^ ^ 


Ci,k —1 


i—/c— 1 




-A 


,A= —1 


Cm,m — l,m—l^m ^ ' O m _i(l;A)r| 


(1 < fc < TO — 1) 
(1 < TO < M), 


A—— 1 


together with the usual balance equations and normalization constraint. 
Proof. Starting with phase 0, we observe as before that, for j > j o + 1, 

n (0J) = 7r (0.io) A 0 E (0,io+l) 


q-'Po 


_ ~J-Jo 

~ n (0,jo) r 0 


and this equality is clearly also valid when j = jo. 

We now proceed by induction. Assuming the result holds for 7To o for 
0 < i < m — 1, i > jo, we have 


n (m,j) — n (m,j 0 ) 'WlEjmjp + i) 


Tf 


m —1 oo 

+ E E 7 r (*,O a *( TO _ *; 

*=0 <?=io+2 


T, 


(m,j) 


m—1 oo 

+ E E 7r (»/) a *< TO ~ *> °) E (m,<)[^(^j)] 

2=0 ^=J 0 + 1 
m—1 oo 

+ EE P(i,i)ai{m - i; 1 )E (m ^ + 1 ) [r ( ^ j) ] 


*=o r=j 0 


m —1 


= ^(m,j 0 ) r O + E C *-0^ 

i—0 




-A 


,A= —1 


j - Oo + 1 ) +1 


„J-JO 


m—1 m—1 oo 

E E c i,uOi( m - *; -f) E 

22=1 2=22 £=Jo + 2 

m—1 m—1 oo 

E E c i,nOti{m -i; 0) 55 

22=1 2=22 £=Jo + l 

m—1 m—1 oo 

E c v'Aa(to - i; 1) 51 

22=1 2=22 £=Jo + l 


’ “ Oo + 1 ) + u \j- 3 o 


r 0 J °E(m,e-i) 


Tf 


hi) 


' - Oo + 1) + u\j_ jo 


0 ”” E (' 


70 


’ - Oo + 1 ) + u \j -30 


T 0 E(m,£+ 1 ) 


ym,3) 


Tf 
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and after applying Lemma 1 and simplifying, we conclude that 


J—JO 

7r (m,j o ) r 0 


+ EE 

k =0 i=fc 
m— 1m— 1 m —1 

+ EEE 


E ai(m-^■A)r 0 , 


j - (jo + 1) + k + 1 


k= 1 u=k i=u 
^ m — 1 m— 1 


E 


k + 1 


-A+l 


r J—Jo 

r 0 


- T— E E c i,kOti{m - i; 1) 

m fe=i i=fc 

i - O'o + 1 ) + k 


cu(m — i\ A )4> m (dim ) 

(1 - r 0 ^ m (a m ))“ +1 - fe 


J - (jo + 1) + k 
k 


j - (jo + 1) + k 
k 


— ^ ] Cm,k 
k =0 


k 


where we have collected terms so that for 1 < fc < m - 1 we have 


m— 1 m—1 


Cm,k — ^ ^ ^ ^ 


u=k i—u 
m—1 


E 1 ai{m-i-,A)(j) m (a m ) A+1 
Lii= _ 1 (1 - r 0 (/)m(am)) u+1 ~ k 


-r- ^2 c iyk ai{m - z; 1) 


i—k 

m—1 


+ ^ ^ Ci k—1 


i=k—1 


El Oii(m — r, A)i 

_A=—1 


-A 


while c m; o — TT(mjo) c m , m — /\__] o/-m —i ( 1 ; A)r t 

claimed. 


-A 

0 


, as 

□ 


3.5. The case where all bases except tm agree. We conclude this section 
by considering the case where ro = r\ = ■ ■ ■ = tm-i 7 ^ tm> as this case 
is satisfied by the Markov chains studied in [30, 29]. The following lemma 
can be used to compute the limiting probability distribution. The proof is 
again omitted, but as with Lemma 1, each formula can be derived using the 
lemmas contained in Appendix C. 

Lemma 2. For a class M Markov chain with all A m ,/x m > 0 and vq = 
r i = • • • = vm -i ~f~ r Mj for each integer u > 0, we have the following three 
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identities: 


• E 

*=jo+2 


^ — (jo + 1) + u\ j_ 3o 


0 1 ) 


= —fijwTo 


1 


1 


rpPM 
1 ( M,j) 


+ y ] ^M r o 


(l-r-o)“ +1 (l-^)“ +1 

1 


F M 


k =0 


(1 _ ro) u+1 ~ k (1 - ft) u+1 ~ k 


3 - (jo + 1) + k 


n J—JO 
r 0 > 


• E 

t=jo + l 


v ~ (jo + 1) + ~£-j 0 


r 0 E ( M ,e) 


rpPM 


= ~^Mro 


(l-r 0 ) u+1 r M (l — ^7t) u+1 


' M 


y n M r 0 


fc=o 


(1 - r 0 ) u + 1 ~ k r M ( 1 - VT7) U+I ~ k 


j - (jo + 1) + k 


' o > 


£ |^-(jo + l) + ^- JOE 


t=i o+i 

= —^M r 0 


r 0 1 ) 

1 1 


T, 


Pm 


(M,j) 


(l-r-o )“ +1 »if(l-^)“ +1 


r J—Jo 
' M 


3 - (jo + 1) + u\ r 3 0 30 


+ y ' qmi" o 


fc =0 


1 


(l-r 0 )“+i- fc r 2 M ( l-^r)“ +1 - fc 


u ) Am 

j - (jo +1) + 


Our next theorem gives an expression for the stationary distribution of a 
class M Markov chain when ro = r\ = ■ ■ ■ = rjvr-i 7 ^ r M- As the proof is 
similar to those of Theorems 3 and 4, we omit the proof. 

Theorem 5. Suppose a class M Markov chain has all X m . p, m > 0 and 
ro = n = ■ ■ ■ = r M -1 + r M - Then, for all 0 < m < M - 1, j > j 0 , 


_ (3 ~ (jo + 1) + k 

7 r (”i.j) “ Z_^ Cm ’ k \ 

k =0 V 

(j ~ (jo + 1) + k 

7T(MJ) — 2_^ ° M ’ k 

k =0 


J-30 
0 > 


r 3~J0 „ J-JO 

r 0 CM,M r M 
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where the {c m . t k}o <k<m<M values satisfy the system of linear equations 


Cm,0 — 7T(m,Jo) 

m — 1 m— 1 

Cm,k — ^m^O ^ ^ ^ ^ Ci,v 


u=k i=u 
m—1 


E 

,A= —1 
1 


A+l 


+ ^ ^ Ci,k— 

i=k — 1 
^ m — 1 

- t— y d : kO!i{m - i ; 1) 

Am 

%—k 

' 1 

Cm,m — Cm—l,m—l^m E dm-i(l; A)ro 

cm, o = CifiiiMro 1 1 


Qi(m. — i; /S)4> m (dm) 

(1 -r o 0 m (a m ))“+ 1 - i: 
E ai(m-i;A))ro‘ 


1-ro 


M — 1 M—1 


+ EE Ci, u ^Mro 

1 


•u=l i=u 


E ai<Af-t;A) 


cu(m - i; A) 

1 


(1 - r 0 ) u+1 


A+i 


d_ifl.)u+i 

V r M ' 


CM,k 


= - E 


i=k 

M — 1 M — 1 


a;(M — i\ 1) 
Am 

+ EE Ci,„flMro 

u=k i=u 

l 


tA+Ri _ r O W+l-fc 
'M V 1 


y a.i(m — i\ A) 


(1 - ro) u+1_ 


(0 < m < M) 


(1 < k < m < 
(1 < m < M) 


(1 < fc < M) 


CM,M = 7T(M,3 0 ) — CM,0, 

together with the usual balance equations and normalization constraint. 


4. Analysis of the M/M/1 /clearing model. In this section we pre¬ 
sent an analysis of the M/M/1/clearing model Markov chain in order to 
prove Theorem 2 (presented in Section 3.2), which we used in the proof of 
Theorems 3 4, and 5. This analysis provides the framework on which the 
CAP method is built. 

Like the ordinary M/M/1 model, the M/M/1/clearing model Markov 
chain (see Fig. 7) has state space {0,1,2, 3,...}, with an arrival rate of 
A = q(j,j + 1) (for all j > 0) and a departure rate of [i = q(j,j — 1) (for all 
j > 2). In addition, all nonzero states in the M/M/1/clearing model have 
an additional transition to state 0 representing a clearing (also known as a 
catastrophe or disaster). All clearing transitions occur with the same rate 
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Fig 7. Markov chain for the M/M/1 /clearing model. For any state j > 0, there is a 
clearing rate with rate a. Note that the transition rate from state 1 to state 0 is fj, + a as 
either a departure or a clearing can cause this transition. The thicker arrow denotes a set 
of transitions. 


a = q(j, 0) (for all j > 2), which we call the clearing rate. Note that from 
state 1, there are two “ways” of transitioning to state 0—a departure or a 
clearing—and hence, g(l,0) = p + a. We observe that each phase, m, of a 
class M Markov chain (for levels j > jo + 1) behaves like an M/M/1/clearing 
Markov chain, with clearing rate 

M 1 

a m = X X a m{i-rrv, A), 

i=m -\-1 A=—1 

except with “clearings” transitioning to a different phase. 

4.1. Preliminary results on clearing models. In this section we present 
two preexisting results from the literature that will aid us in proving The¬ 
orem 2. Our first result gives the limiting probability distribution of the 
M/M/1/clearing model: see e.g., Corollary 4.2.2 of [1], as well as Exercise 
10.7 of [12]. 

Lemma 3. In an M/M/1/ clearing model with arrival, departure, and 
clearing rates X, ft, and a, respectively, the limiting probability distribution 
is given by 

Kj = (1 - /3</(a))(p</(a) J ), 

where p = X/p and </(•) is the Laplace transform of the busy period of an 
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M/M/1 system: 


<l>(s) 


s T- A T- /i — -^/(s + A + p) 2 — 4A/i 
2A 


Proof. See the proof of Corollary 4.2.2 of [1]. 


□ 


The next result is also known, and gives an expression for a probability 
that is useful in computing values of the form E^ T/ 4 in the M/M/1 clearing 
model. A similar result, presented in the context of Brownian motion, is 
given in Problems 22 and 23 from Chapter 7 of [15]. 


Lemma 4. In an M/M/1 /clearing model with arrival, departure, and 
clearing rates A, /a, and a, respectively, the probability that one reaches state 
j > 0 before state 0, given that one starts in state £ > 0, is given by 


P£-*j 


(p(f(a)) j /1 - (p<f>(a) 2 Y) 
1 - (p4>(a) 2 )i 
4 J 


if £ < j 
'iff- > j- 


Proof. First, note that in the degenerate case where £ = j, we are 
already at state j from the start, and so we reach state j before reaching 
state 0 surely , yielding pi^j = 1. Substituting £ = j in either branch of the 
claimed expression for Pe->j yields 1, validating the claim in this case. 

Next, we consider the case where £ > j, which will be the simpler of the 
two remaining cases. In this case, Pi^j can be viewed as the probability 
that the sum of £ — j independent M/M/1 busy periods (without clearing), 
B\, L> 2 ,..., Bc_j, do not exceed the exponentially distributed “clearing” ran¬ 
dom variable ( a : 


Pt-+j 



= E 




<f(a) j - e 


as claimed, with the next-to-last equality following from the alternate inter¬ 
pretation of the Laplace Transform (see Appendix A for details). 

Now let us consider the remaining case where £ < j. In this case, it will 
be helpful to consider two Poisson processes, one associated with arrivals, 
occurring with rate A, and the other associated with departures, occurring 
with rate p. Departures can happen even at state 0, although at state 0 
departures do not cause a change of state. Let N^(t) and Nr>{t) be the 
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number of such arrivals and departures during time interval [0, t], assuming 
that we are in state t at time 0. 

Next, let To = inf{f: £ + — Njj(t)) = 0} be the first time after 

0 until we have t departures in excess of arrivals, and let Tj = inf {i: l + 
( N A (t ) — Njjfjt)) = j} be the first time after 0 until we have j — £ arrivals in 
excess of departures. Although there may be positive probability that one of 
of these two events may never happen (i.e., max{ro, Tj} = +oo), at least one 
of these events will happen almost surely. Moreover, if either of these events 
happens before a clearing, which will occur at time Ca ~ Exponential(a) 
(independent of both To and Tj), then to and Tj describe the first time that 
we will reach state 0 and j, respectively. 

Given this notation, we can express pe^j, the probability that one next 
reaches state j > £ before state 0 in an M/M/1/clearing model, given that 
one starts in state £ > 0, by 

Vl-+j = < min-Jro, (a}) = E[e~ aT ° ■ I{t 0 < Tj}], 

where /{•} is the indicator function. Similarly, if we let pe-/>j be the prob¬ 
ability that we reach 0—via departures, rather than via a clearing—before 
reaching j and before a clearing, we have 

Pe-frj = f (fo < min{Tj,Ca}) = E[e' QT t • I{tj < t 0 }]. 

At this point, it will be useful to compute the quantities E[e _ “ T °] and 
E[e _QT) ']. Observe that To is the time until we first have £ departures in 
excess of arrivals. We can think of each time “departures minus arrivals” 
increments by one as the completion of an M/M/1 busy period. Hence, To 
corresponds to the time until we have completed l consecutive independent 
busy periods. Meanwhile, Tj is the time until we first have j —1> 0 arrivals 
in excess of departures. Just as we can think each time “departures minus 
arrivals” increments by one as the completion of an M/M/1 busy period, we 
can also think of the each time “arrivals minus departures” increments by 
one as the completion of an M/M/1 busy period where we think of arrivals 
as “departures” occurring with rate A and departures as “arrivals” occurring 
with rate /i. Hence, Tj corresponds to the time until we have completed j — l 
consecutive independent busy periods with arrival rate p and departure rate 
A. Consequently 

E[e“" r °] = E[e- aT ^] = Tj(a) j ~ e , 


where and rj(-) are the Laplace transforms of the M/M/1 busy peri¬ 
ods with arrival and departure rate pairs (X, p) and (p, A), respectively. We 
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observe that for all s > 0, 


V(s) = 


s + p + A — y/ (s + p + A) 2 — 4/iA 


A\ / s + X + p — ^/(s + A + p)^ — 4A/r 
l 2A 


= P0(s). 

Note that in the case that p ^ 1, we must have ??(0) ^ 0(0), and in particular 
one of these transforms will not evaluate to 1. This is not a problem as if p < 
1 (respectively, p > 1), the underlying random variable of q (respectively, 
0) has positive probability mass at infinity, and will thus not satisfy the 
“usual” condition of Laplace transforms evaluating to 1 at 0. 

We proceed to use these expectations to determine pe->j : 

0(a)* = E[e- QT0 ] 

= E[e- Qro • /{r 0 < Tj}\ + E[e _ “ T ° • I{ Tj < r 0 }] 

= E[e _Qr ° • /{ t 0 < Tj}\ + 4>(a) j ■ E[e _QTj ' • I{tj < t 0 }] 

= Pe^j + ^(aYpe^j, 

{p^{a)y- 1 = E[e~^] 

= E[e- Qr > • I{t 0 < Tj}\ + E[e-“^ • I{ Tj < r 0 }] 

= ( p4>(a)y • E[e"" T0 • 7 {t 0 < Tj}] + E[e -Qr ? • I{tj < r 0 }] 

= (/30(a))- 7 (Pw) +Pe^j- 

We justify E[e - " ro • I{tj < To}] = 0(a) J • E[e~ Qr > • I {t, < to}] by observing 
that given that Tj < to, we reach state j before state 0 (ignoring clear¬ 
ings), so we can only reach state 0 by performing j consecutive busy periods 
after reaching j. We justify the analogous equality K[e~ aTj ■ I{tq < T j}\ = 
( p(f)(a)y •E[e _aT ° -/{to < tj}] by observing that given that To < Tj, we reach 
state 0 before state j (ignoring clearings), so we can only reach state j by 
performing j consecutive “busy” periods in an M/M/1 model with arrival 
rate p and departure rate A. 

We now have a system of two linear equations in the two unknowns, pi->j 
and Pe-frj- Solving the system for Pe^j and simplifying, we find that 


Pt-*j ~ 


(.p0(q)) J e - (p0(q)) J 0(a) 

1 - (p0(a))J'0(a)-? 


(p0(a)) J *(1 - (p0(a) 2 ) £ ) 
1 - (p0(a) 2 y 


which proves the claim. 
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4.2. Applying clearing model analysis toward proving Theorem 2. We 

in an M/M/1/clearing model, 


rpA 

3 


now use Lemmas 3 and 4 to compute E^ 
where A is the set of nonzero states. This result is presented in Lemma 5. 
Finally, we will recast Lemma 5 in the context of class M Markov chains, 
allowing us to prove Theorem 2 from Section 3.2. 

Lemma 5. In an M/M/1 /clearing model with arrival, departure, and 
clearing rates A, p, and a, respectively, if A = {1,2,3,...} denotes the set 
of nonzero states of the state space of the underlying Markov chain, then 


E, [Tj] = 


(p/{u)) J £+1 (1 ~ (p/(a) 2 Y) 

A(1 - p4>(a) 2 ) 
p/(aY^ +1 (1 - {pcfja) 2 /) 

A(1 - pf>(a) 2 ) 


ift < j 


if t > j- 


Proof. We first consider the case where £ < j. We claim that 
(4.1) Er [T/] = ( Pl ^)E, [T/] , 

recalling that is the probability that one reaches state £ before state 
0 given initial state 1. Equivalently, in our setting, we may interpret pi->g 
to be the probability that one reaches state £ before leaving A, given initial 
state 1, as 0 is the only state not in A. The claim in Equation (4.1) follows 
from conditional expectation and the fact that given that we start in state 
1, we either 

• reach state £ before leaving A, in which case the the expected cumu¬ 
lative time spent in state j before leaving A is E^ T/ 4 (note that no 
time is spent in j before reaching £, as £ < j), 

• or we do not reach state £ before leaving A, in which case we also do 
not reach state j, and hence we spend 0 time in state j before leaving 
A. 

From Lemma 4, we know that for £ < j, we have 


(4.2) 


PtM 


(p/(a)) j £ (1 ~ (p/(a) 2 Y) 

1 - ( p<f>{a) 2 Y 


Hence, in order to determine E^ 


from Equation (4.1), we need only de¬ 


termine Ei 


Ti 


We compute this quantity via the renewal reward theorem. 
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Let us earn reward in state j at rate 1, and consider a cycle from state 0 until 
one returns to 0 again (after leaving 0). We also use the fact from Lemma 3 
that the limiting probability of being in state j in an M/M/1/clearing model 
is given by (1 — pcf>(a)){pcj)(a .)) 3 . Hence, by the renewal reward theorem, we 
have 


(4.3) 


Ei 



E [Be 

+ 1/A 


(1 - /?</>(«))(^(ct))- 7 , 


where Be denotes the busy period of an M/M/1/clearing model. To de¬ 
termine E[L>c], observe that Be = min{R, I/*}, where B is an independent 
random variable distributed like the busy period of an M/M/1 model without 
clearing, and Q a ~ Exponential(a) is an exponentially distributed clearing 
time. Taking the expectation, we have 


/*oo roc 

E [Bc\ = E[min(L>, £a)] = / P(R > t)P(C<* > t) dt = / E(R > t)t 

Jo Jo 


—at 


dt 


1 

- P(B> t ) ( ae~ at ) dt = 

o 


P(B >Ca) 1 - P(H < C a ) 


a 


a 


1 — 4>(a) 


a 


where the final step follows from an alternate interpretation of the Laplace 
transform (see Appendix A for details), noting that (/>(•) is the Laplace trans¬ 
form of B. 

Returning to Equation (4.3), we find that 


(4.4) E, [T/] = (LAM + i) (1 - p4,( a ))(p4,( a ))i = M/M 
where we make use of the identity 


LAM + I ) (1 - p^a)) 

a A 


1 

A 


in our simplification. This identity can be verified algebraically by using 
the explicit form of 4>(s). Alternatively, let Eo[To] be the expected duration 
of time spent in state 0 in a cycle starting from state 0, and ending with a 
return to state 0 from a nonzero state. Then by the renewal reward theorem, 


E 0 [T 0 ] = (E [B C ] + £) (i - M«)) = + £) (! - (xK*)). 












32 


S. DOROUDI, B. FRALIX, M. HARCHOL-BALTER 


We can also observe that during such a cycle, the only time spent in state 0 is 
during the initial residence, as a revisit to state 0 ends the cycle, so Eo[7o] = 
1/A. Setting these quantities equal to one another yields the claimed identity 
directly. 


We proceed to use Equation (4.1) in determining E^ 
where £ < j), by substituting in values from Equations (4 


(in the case 


.2) and (4.4): 


e, [: if] = 


Ei 




3 J _ = ( ( P ^)) 3 


Pl->i 


V A 


1 - (pcj)(a 


, 2 \l 


{p4>(a)) e X (1 - pc/){a) 2 ) 


= (p<t>(a)) 3 m (1 - (p<j>(a) 2 ) e ) 

A(i — p4>( a ) 2 ) 

Next, we consider the case where £ > j (note that the two branches 
in the claimed expression coincide when £ = j). We again use conditional 
expectation, this time obtaining 

p0(a)(l - {pYjafy) 

A(1 — p4>( a ) 2 ) 


E, [Tf] = (?<-,■)% [Tf] = 

pY(aY^ +1 ( 1 - (p4>(a) 2 ) J ) 


A(1 - pY(a) 2 ) 

which completes the proof of the claim. Note that we have obtained E j Tj 
by substituting £ = j into the expression for E^ Tj 4 j, which we found for 

£ < j, and we have also used the fact from Lemma 4 that Pe^-j = 4>(aY~^ 
whenever £ > j. □ 

Finally, we use Lemma 5 to prove Theorem 2. 

Theorem 2 . For any class M Markov chain, if \ m , p m > 0 and £, j > 
jo + 1, we have 


(4.5) E( m ^) 


(m-i) 


_ f n m r 3 m. e (l - (r m (/> m (a m )) f jo ) if £ < j 

1 Q, m (j)m(oim) e ~ j (l - {r m (f m (a m )) j ~ jo ) if £ > j- 


Proof. Observe that the time spent in state ( m,j ) before leaving phase 
m, given initial state (m, £) in a class M Markov chain with A m , p m > 0 
is stochastically identical to the time spent in state j — jo before reaching 
state 0, given initial state £ — jo in an M/M/1 /clearing model with arrival, 
departure, and clearing rates A m , p m , and a m , respectively. That is, 


E 


(m,£) 


' (rn,j) 


= E £ [t/ 
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where the quantity on the left-hand side is associated with the class M 
Markov chain, and the quantity on the right-hand side with the M/M/l/clear- 
ing model Markov chain (with the appropriate transition rate parameters 
and A = {1, 2,3,...}). 

We proceed to complete the proof by applying Lemma 5. Recall when 
Mm > 0, we have the notation p m = A r m = p m 0m(«m), and 
T m /(A m (i — r rn (f) m (a rn ))). Applying Lemma 5 when £ < j yields 


®(m,£) 


T’Ai 

1 ( m J) 


(pm^m^m))^ jo) ^ jo)+1 (j- ~ j °) 

Am (1 Mm0m(^m)^) 


r J m M (l - (r m 0m(<am))^ jo ) 
Am (1 ^m^mt^m)) 



while when £ > j, Lemma 5 yields 




n^Pm 
1 ( m,j ) 


J °) jo)+1 (l ~ ( j °) 

Arrt(l Pm^mi^mY) 
r m (t>m{(y-raY^ j (l ~ 

Am(l T'm ( t > m((Xm)) 

tt m 4>m(a m y~ j (l - (r m (/)m(a m )) j ~ jo ) , 


as claimed. 


□ 


5. Extending the scope of the CAP Method. In this section we 
briefly touch upon ways in which the CAP method can be extended beyond 
class M Markov chains. 

5.1. Chains with “catastrophes”. Recall that the M/M/1 clearing model 
is used to model a system where there can be a catastrophe from any nonzero 
state causing an immediate transition to state 0. Similarly, we can consider 
a modification of a class M Markov chain where from any state ( m,j ) with 
j > jo + 1, a catastrophe can occur taking one to state x € Af with rate 
a m (x) = q((m,j),x ). 4 That is, each phase can have several catastrophe 
rates, one for each state in the non-repeating portion. In this case, it will be 
useful to redefine a m as follows: 

M 1 

= ^2 a m (x) + a m (i-m; A). 

x£Af i=m- 1-1 A=—1 

4 Whether or not catastrophes can also occur in states ( m,jo ) will not change the 
analysis as arbitrary transitions from states (m,j o) to states x G Af are already allowed 
in class M Markov chains. 
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The CAP method can easily be modified to give limiting probabilities for 
these types of Markov chains. 

5.2. Skipping levels when transitioning between phases. Although the as¬ 
sumption that transitions from state ( m,j) to state (m,£) can only occur 
only if £ = j ± 1 is essential to the CAP method, the assumption that 
transitions from state (m,j) to state (i,£) (where i > m) can only occur 
if £ = j A 1 is much less important. That is, the CAP method may be ex¬ 
tended to allow for nonzero transition rates of the form a m (Ai;A 2 ) with 
d < A 2 < D for some d,D € Z. However, it is advisable to treat the levels 

Tjq,Tjq_)_^, ..., Tyj+rna.xjl^ D \}_i special cases, just as was treated as 

a special case in the analysis presented throughout this paper. 

5.3. Chains with an infinite number of phases. Consider a chain with 
the structure of a class M chain, except with infinitely many phases (i.e., 
m £ {0,1,2,...}), and a possibly infinite non-repeating portion, A f. The 
CAP method may be used to determine the {c mj k}o<k<m values in terms of 
{'K x \x&N for the first K phases by solving a system of at most 0(K 2 ) equa¬ 
tions. This is because the CAP method provides recurrences such that each 
{c m ,k}o<k<m value can be expressed in terms of {ci } k}o<k<i<m-i values; that 
is, only information about lower-numbered phases (and the non-repeating 
portion) is needed to compute each c m ^. We can first express such values 
for phase m = 0, then phase m = 1, and so on. Once these values—along 
with the easily determined corresponding base terms—have been obtained, 
we can use the CAP method to find the limiting probabilities for all states 
in the first K phases as long as we know the {ir x }x&j\[ values. 

Such a procedure is typically not useful, as the {'K x }x&M values are usually 
determined via the normalization constraint, which requires expressing lim¬ 
iting probabilities, 7r( m j), in terms of {'K x }x&N for all phases, rather than for 
only the first I\ phases. However, there are settings where sufficient infor¬ 
mation about the structure of {Tr x } x eAf may be obtained via other analytic 
approaches, allowing for the CAP method to compute the limiting prob¬ 
ability of the first K phases (where K can be as high as desired, subject 
to computational constraints). For example, a two-class priority queue can 
be modeled by an infinite phase variant of a class M Markov chain. In that 
setting, queueing-theoretic analysis provides sufficient information about the 
structure of the limiting probabilities in the non-repeating portion (see [26] ), 
making the CAP method a useful tool for that problem. 

6. Conclusion. This paper presents a study of the stationary distri¬ 
bution of quasi-birth-death (QBD) continuous time Markov chains in class 
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M. Class M Markov chains are ergodic chains consisting of a finite nonre¬ 
peating portion and an infinite repeating portion. The repeating portion of 
a class M chain consists of an infinite number of levels and a finite num¬ 
ber of phases. Moreover, transitions in such chains are skip-free in level, in 
that one can only transition between consecutive levels, and unidirectional 
in phase, in that one can only transition from lower-numbered phases to 
higher-numbered phases. Despite these restrictions, class M Markov chains 
are used extensively in modeling computing, service, and manufacturing sys¬ 
tems, as they allow for keeping track of both the number of jobs in a system 
(via levels), and the state of the server(s) and/or the arrival process to the 
system (via phases). 

This paper develops and introduces a novel technique, Clearing Analy¬ 
sis on Phases (CAP), for determining the limiting probabilities of class M 
chains exactly. This method proceeds iteratively among the phases, by first 
determining the form of the limiting probabilities of the states in phase 0, 
then proceeding to do the same for the states in phase 1, and so on. As 
suggested by its name, the CAP method uses clearing model analysis to 
determine the structure of the limiting probabilities in each phase. 

Unlike most existing techniques for solving QBDs, which rely upon the 
matrix-geometric approach, the CAP method avoids the task of finding the 
complete rate matrix, R, entirely. Instead, the CAP method yields the limit¬ 
ing probabilities of each state, (rn, j), in the repeating portion of the Markov 
chain as a linear combination of scalar base terms (with weights dependent 
on the phase, m), each raised to a power corresponding to the level, j. These 
base terms turn out to be the diagonal elements of the rate matrix, R. The 
weights of these linear combinations can be determined by solving a finite 
system of linear equations. We also observe that the structure of the weights 
of these linear combinations can depend on the multiplicity structure of the 
base terms. 

The CAP method can be applied to Markov chains beyond those in class 
M, as discussed in Section 5. For example, the CAP method can be used to 
determine limiting probabilities in chains where one or more phases allow for 
immediate “catastrophe” transitions to states in the non-repeating portion. 
As another example, the CAP method can also be applied to Markov chains 
where transitions between phases can be accompanied with a change in level 
exceeding 1. The CAP method can also be used to study some chains with 
an infinite number of phases. There is ample room for future work to extend 
the CAP method in a variety of directions. 

The CAP method and the solution form it provides offer several impactful 
advantages. First, while many existing methods for determining the limiting 
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probabilities of QBDs exploit the relationship between successive levels, the 
CAP method exploits the relationship between successive phases, thereby 
offering complementary probabilistic intuition on the structure and steady- 
state behavior of class M Markov chains. This method also provides an 
additional tool for practitioners who are studying systems that can be mod¬ 
eled by class M Markov chains. Depending on the application domain, the 
scalar solution form of the CAP method may have advantages over other 
solution forms for computing certain metrics of interest (e.g., mean values, 
higher moments, tail probabilities, etc.). While this paper does not cover 
using the solution of the CAP method to derive metrics of interest, as such 
metrics are often application specific, we hope that future work can find 
novel uses for the CAP method in a variety of settings. 
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Appendix 

APPENDIX A: AN ALTERNATIVE INTERPRETATION OF THE 

LAPLACE TRANSFORM 

Let X be a nonnegative random variable, with well-defined Laplace trans¬ 
form ip(-) (i.e., "i/> is defined on all positive reals), cumulative distribution 
function, Fx(-), and probability density function, note that X may 

have nonzero probability mass at +oo, in which case J 0 °° fx(t ) dt < 1 (where 
we interpret the integral as being evaluated on {t £ M: 0 < t < oo}). Then 
for any constant w > 0, we have the following interpretation of ip: 

poo 

ip(w) = / e~ wt f x (t)dt 

Jo 

poo 

= e~ wt F x (t )\~ + / F x (t) {we~ wt ) dt 

Jo 

= P{v < C4, 

where ( w ~ Exponential^) is a random variable independent of X. 


APPENDIX B: THE COMPLETE PROOF OF THEOREM 3 


Proof. We prove the theorem via strong induction on the phase, m. 
Specifically, for each phase m, we will show that 7T( m ,j ) takes the form 

7T(m,j) = Yk=o c 'm,,k' r i^ 30 for all j > j 0 + 1, and show that {cm,k}o<k<m-l 
satisfies 


rkVm / 

771-1 1 A 

Y Y. c i,kOti{m - i; A)rf 

i=k A= — 1 / 

| if r m ,r fe > 0 


A mij'k ^m)( 1 0771 (^tti)^) ^ 

Cm ,k — < 

y; y c^ k OLi{m-i]/S)rk 

i=k A= -1 


if Tk > r m = 0 

if r k = 0, 


/^?7i(l 'Ck) Otm 

0 



while Cm^m = ^o~Yk=o c m,k- Finally, after completing the inductive proof, 
we justify that the remaining linear equations in the proposed system are 
ordinary balance equations together with the normalization constraint. 


Base case: 

We begin our strong induction by verifying that the claim holds for the 
base case (i.e., for m = 0). By the ergodicity requirement on class M Markov 
chains, Ao > 0, leaving two sub-cases when m = 0: the case where /io > 0, 
and the case where = 0- In the first case, where /j,q > 0, Equation (3.3) 
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yields 


E 


(OJo+l) 


rpPo 

J (0j) 


= fVo 10 X (1 - ^o^o(ao)) = 


r. 


,3-30 

o_ 

An 


Now consider the other sub-case, where /ig = 0, recalling that in this 


rpPo 

>,j) 


for this case, 
ecrease the level, as 


case, we have ro = Ao/(Ao + ao)- We calculate E( 0 ,j 0 +i) 
by noting that transitions within states in Pq cannot c 
follows: starting at state (0, jo + 1), we either never visit state (0, j) before 
leaving Pq, or we visit state (0, j) exactly once before leaving To- The latter 
occurs with probability 


Aq 


Aq + ol o 


3-3 0-1 


= r; 


3 - 30-1 


in which case, we spend an average of l/(Ao + cto) = ^o/Ao units of time in 
state (0, j). Hence, we find that 


E 


(Ojjo+l) 


rpPo 


— J-jo- 1 
~ r 0 


Ao 


which coincides with our finding for the case where > 0. 
In both cases, applying Theorem 1 yields 


7r (0j) = 7r (o,io)AoE( 0 ,j 0 +i) 


rpPo 

1 (ojy 


= n (O,j o ) X 0 


r\ 


. 3-3 0 
0 


7r (0jo) r 0 J ° 


7 — 7 0 

= Co, 0^0 > 


where co,o = ^(ojo)- Hence, 7i7 0 j) takes the claimed form. Moreover, co,o 
satisfies the claimed constraint as co,o = ^(ojo) ~ c m,k = 7T(oj 0 ) — 0 = 

7T(oj 0 ), because the sum is empty when m = 0. Note that when m = 0, 
{cm,it}o<fe<m<JW is empty, and hence, there are no constraints on these values 
that require verification. 

Inductive step: 

Next, we proceed to the inductive step and assume the induction hypoth¬ 
esis holds for all phases i € {0,1,..., m — 1}. In particular, we assume that 
7i7jj) = ELo c i,fe r fc _ " , ° f° r * < m. Tor convenience, we introduce the 
notation 


T m,j — A m E( mj[J+1 ) 




and Vm.kj 




£=l 


ppPm 
1 ( m,j ) 
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Using this notation, we apply Theorem 1 and the induction hypothesis, 
which yields 5 


m — 1 oo 1 


n (rn,j) ~ ' 7r (m,j 0 )AmE( mj j 0 _|_i) ^ ^ 7T(i,£- A)OLi(m l\ A)E( m ^) 

i=0 £=1 A= — 1 

m— 1 oo 1 / i 

= K(.mJo) r m,j + J2J2 J2 ou{m-i;A) ( J2 c i,k r l~ J0 ~ A ^(m,e) 


i =0 1=1 A=-l 


k =0 i=k 


= V,io) T ™,i+ 1] 1 ] ( Ci.fc 1 ] ai(m-i-,A)r£ ) ( J0]E W) [ T ^m,j)] 


(B.l) 


m— 1m— 1 

~ 7r (m,jo)^"T'0’ + 

fc=0 i=k 



^m,k,j • 


We proceed to compute T m j and 'hm.fcj separately in the following cases: 

• Case 1. \rn.i l^m A 0 

• Case 2: A m > = 0 

• Case 3: /r m > A m = 0 

• Case 4: /r m = A m = 0 


Computations for Case 1 (A m ,/r m > 0): 
When Xm,Hm > 0, Equation (3.3) yields T mj - 


A m E 


(mj'o+l) 


n-’Pm 

± {rn,j) 


5 Note that X^a=-i a i( m ~ *; A}r() i s n °t well-defined when r*, = 0, as 0 _1 and 0° are 
not well-defined. However, this is just a convenient formal manipulation which will remain 
true if we assign any real value to X^a=_i a i{ m ~ *; A}rjj) as \f r m ,k,j = 0 in the Tk = 0 
case, and the “contribution” to the sum by an index k such that rj, = 0 is also 0. One 
can verify that this is “harmless” by examining such k indices in isolation. Note further 
that we have also used the fact that 7r( ij0 ) also satisfies the claimed form for all i < m, 
which is true as Cj,i = 7r (ij 0 ) — c »,fc (from the inductive hypothesis) implies that 

7r (*o'o) = Sfc=o Ci < k = Sfc=o c h^ r fc> except that once again values of r*, = 0 yield undefined 
quantities of the form 0°. Once again, this is a convenient formal manipulation that will 
not affect our results if we simply assign 0° = 1 in this context. 
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rin J0 • We also find that 
00 

t=\ 

j 00 

E ^-JOip IjiPm 1 , \ " t-jOTff \rpP m 

r k ^(m/) + r k ^O/) [ J (mj) 

*=j'o+i *=j+i 

= f ^ r k~ j ° r m i (l “ (rm<j>m(oi m )Y~ jo ^ 

V=io+l 

OO \ 

+ ^2 r e k ~ jo ^ m (a m ) e ~ j (l - (r m i(a ra )) j “ jo ) I 

l=j+i J 

_ r k r m (r{~ jo - r J m jo ) 

A m(j"k ( f > m{c >i m)'l"k) 

where the last equality follows from well known geometric sum identities. 
Note that this expression is well-defined because r k 7 ^ r m by assumption 
and r m (j)m{a m ) / 1 . 

Computations for Case 2 (X m > /j„, = 0): 

When A m > [i m = 0, we recall that r m = X m /(X m + a m ) and compute 
E( m ^) T^mj) as f°ll° ws: starting at state (m,£), we either never visit state 
(■ m,j ) before leaving P m , or we visit state (m,j) exactly once before leav¬ 
ing P m . If £ > j, we never visit state (m,j) before leaving P m (and so 
E( m ^) T^mj) = ®)> but if £ < j, we visit state (m,j) exactly once before 

leaving P m with probability and this visit will last an average time of 
1/(A m + %) = rm/X m , yielding 



In particular, = A m E( mj - 0+1 ) = rin J0 , coinciding with the ex- 
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pression for Y mj from Case 1, and furthermore, we have 


'I' 


m,k,j 


E 

i=jo+l 

j 

E ; 

«=J 0+1 




T, 


(m,j) 


t—J o 


E 


(m,£) 


T, 


( m,j) 


+ E 

t=i+i 


rjfe J0 E(m,<) 




E ■'* 

«=io+i 

Ffc?’ m (r 


^-ioE-^+i 


j-jo _ r j-jo-] 
1 m ) 


rkr m {r 


,3-JO _ r J~30\ 
1 m ) 


A/n Ok r m ,) 


^m(j"k F m ,)( 1 4 > m{p , -rn)'f'k) 


which coincides with the expression for ^ m: k,j that we found in Case 1. 
The last equality follows by noting that in this case we have (p m (s) = 0, and 
hence 1 - </> m (a m )r fc = 1. 

Computations for Case 3 (/i rn > X m = 0): 


When n m > \ m = 0, we have T m j = A m E (mj - 0+1) 


‘( m b) 


= 0. Next, we 


compute E( m a 


‘ (m,j) 


as follows: starting at state ( m , £), if £ < j, we never 


visit j before leaving P m , while if t > j we will visit j exactly once with 
probability /Jm/ (p m + a mY^ J and this visit will last an average duration of 
1 /(nm + a m ) units of time. Consequently, E( m ^ \t, 
case and 

" nP„ 


( m,j ) 


= 0 in the former 


E 


( m,i) 


T, 


(m,j) 


e-j 

Hm 


{fJ-m + OtmY ■ ?+1 


in the latter case. Finally, we have 




m,k,j 


T, 


J2 r k J ° E (m,e) 

i=jo+l 

E hi 

^=io+i 


(mj) 


(m,j) 


+ E 

£=i 




T, 


(m,j) 


J-jo ..1-3 
k r jm 


J)-30 


^ (/x m + a m ) £ ^ +1 // m (l - r fc ) + a r 


Computations for Case 4 (/j m = A m = 0): 



CLEARING ANALYSIS ON PHASES 


43 


When n m = X m = 0, we again have T mj - = A m E (mj0+1 ) 


T, 


(raj) 


= 0, as 


in Case 3. Next, we compute E( m ^ 


T, 


( m,j) 


as follows: in this case any visit 


to P m will consist entirely of one visit to the initial state in P rn . as there are 


no transitions to other states in the same phase. Hence, E( m ^ 


(raj) 


— 


if 1 = 3, and E (m ^) 


(raj) 


= 0 otherwise. Consequently, 




m,k,j 


J2 r k 


e=i 


'T'Pm 
1 (m,j) 


= rl~ JO E 


(raj) 


n^Pm 
1 (raj) 


J~i o 

r k _ 

Mm( 1 - r k ) + a m ’ 


r j-j o 
[k 


which coincides with the expression for 1 I' m ,fcj that we found in Case 3. The 
last equality follows by noting that n m = 0, and hence // m (l — r k ) = 0. 

Completing the inductive step: 

We now proceed to substitute the results of our computations into Equa¬ 
tion (B.l). Since Y m ,■ can be given by the same expression for both Case 1 
and 2, and the same holds for we consider these two cases together, 

and note that they jointly make up the case where r m > 0. For j > jo + 1, 


TpmJ) 7T(m, jo) YmJ 4“ A A ) ( ^ ( CXi(jTl i] J 

k=0 i=k 


m — 1 m — 1 


A= —1 
1 


7T(m,j 0 ) r ™ 30 + Y Y [ Ci ’ k Y Ui(m-i;A)r£ 


= J2 c ™’ kr k 

k =0 


r, r ( r J ~ Jo — rJ—JO \ 

• k> m\i fc _ ' m J 

A mij'k ^m)( 1 0m(c^m)rfc) 


where we have collected terms with 

/ m—l 1 


TkTm 


Y Y a, k ai{m - i; A)r£ 


Cm,k — 


k i=k A——1 


A mij'k r m )( 1 


(0 < k < m < M: r m , r k >0) 


and c m , fc = 0 when r m > r k = 0 and c m , m = 7r (mj0 ) ~ YJk=o °m,k, as 
claimed. 

The expressions for Y mj - and '& m ,k,j also coincide across Cases 3 and 4 
(although they are distinct from their Case 1 and 2 counterparts), so we also 
consider these two cases together, noting that they jointly make up the case 
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where A m = r m = 0: 

m— 1m—1 

/ 1 


EE 

Q,fc XI 


•11 

O 

II 

-Sd 

V A=—1 

m—1 m— 1 / 

' i 



c i,k ^2 

ai(m — i; A)?’^ 

•11 

O 

II 

, A=—1 


m 

= X c m,kri~ JO 

k =0 




A)r k T m,k,j 


r 


j-jo 


A^m(l Tk) ~ 1" 


where we have collected terms with 


m —1 1 

X] X c i,k&i{m - i] A)r£ 

c m ,k = — tt - 7 T—- (0 < k < m < M : r m , r k > 0) 

/i, n (l r k ) T- a m 


and c m y. = 0 when r m = r k = 0. Observe that since r m = 0, it appears that 
we can allow c m:Tn to take any real value, so in order to satisfy the induction 
hypothesis, we set c m , m = 7i7 mj0 ) — Y?k=o c m,k in the r m = 0 case as well. 
Also note that we have set c m j. = 0 when r k = 0 in both the r m > 0 and 
r m = 0 cases. This completes the inductive step and the proof by induction. 

The balance equations and normalization constraint: 

The equations with 7T( mj0 ) and ir x in their left-hand sides in our proposed 
system are ordinary balance equations (that have been normalized so that 
there are no coefficients on the left-hand side). 

It remains to verify that the final equation, which is the normalization 
constraint: 


M M oo 

1 = X 7Tx X + X X f ( m J) 

x£j\f m =0 m=0 j=jo+l 

MM M m— 1 oo 

= X ^ + 2 £ Cm ’ k + c m,kr{- ]o 

x&Af to=0 k= 0 m=0 k= 0 i=io+l 


M 777 


- X nx + ^2 ^2 

xeAf m =0 k =0 


^m,k^k 

1 - r k ' 


□ 
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APPENDIX C: NEGATIVE BINOMIAL LEMMAS 

These lemmas are used to derive our main results, and are likely known, 
but to make the paper self-contained we both state and prove them. 

Lemma 6. For each (3 £ (0,1), we have 



P 

(1 - / 3 ) n+1 ' 



The next lemma shows how to compute a truncated version of the above 
series. 

Lemma 7. For f 3 ^ 1, we have 
y- ft - Jo + n \ 1) = P ~ P 3 ~ {:,0 ~ 1) 

£-;V n r (i— p) n+i 

|Yj ~ Jo + k \ /j - jo + k - 1\ 1 

[\ k ) \ k-1 )\ (1 -0)"+i-*‘ 
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Proof. Starting with the left-hand-side, we have 


3 -1 

E 


■ ~ i° + n ] pt-Uo-v = 


3-1 l 

EE 

t=jo X=jo 
3-1 3-1 

EE 

x=jo i=x 
3 — 1 

E 


x - jo + n — 1 
7i—l 


C?o—!) 


x - jo + n - 1^ ^-(j 0 -i) 

77 — 1 


jo + n 1 | o®—O'o—!) 


77—1 


i-i 

>E^ 


j-i 


(i - /?) 5 


i j ~ i 

= (T“E) E 

®=jo 


x jo + n 1 ] / 3 a: _(' , 0 -i)(l — 
n — 1 


x jo + ^ 1 l Go—i) 

n — 1 


1 - 


1 J ~ 1 

(TEs) E 

a:=jo 


1 / j ~ jo + n - 1 

n 


oi—Uo—i ) 


* JO + W 1 |oa:_(j 0 -l) 

n — 1 


1 - 


j - jo + n 
n 



j-jo+n-1 
n — 1 


Setting 


j-i 

a " = E 




n 


bn — 


=30 

j - jo + n 
n 


j ~ Jo + n - 1 
n — 1 




we see that for each n G {1,2,3,...} we have 

1 b n 


CLn. — 


where 


a o — 


1-/3 1-0 

0 - 0J-CJO- 1 ) 


1-0 

The solution to this recursion is given by 


CLn — 


ao 


E 




(1 - 0) n (1 - /3) n+1 - fc 


P 


>3-(30-l) 
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or, equivalently, 


1 _ ” 
a n = — E 

k =l L 


3 ~ 3 o + n 
n 


j ~ jo + n - 1 
n — 1 




(i — /3) n+1 n / V n-i /J (i - /3) n+1_fc ’ 

which completes our derivation. □ 

The next lemma can be viewed as a generalization of Lemma 6. 

Lemma 8. For (5 E (0,1), 


V [ L ~ Jo + n \ „i-j 1 , y' 

1 n JV (! — /3)n+l ' Z-< 


3 ~ jo+ k 
k 


3 ~ jo + k - 1 
k — 1 


— R\ri+l — k 


(i -P) 


Proof. The key to deriving this series is to use both Lemmas 6 and 7. 
Here 


E 


i - jo + n\ t-. 


-J _ 030—3 


E 


-/S' 


(lo-l)-i 


t - jo + n \ e _ 


3~ 1 

E 


+ n i^-Oo-h 

n 


+°- 


(1-/5)"+! 

n 

+E 


-/S' 


(jo-H-i 


P — P' 


\3-Uo~ 1 ) 


(1-PY+ 1 


j -jo+ k\ _ j -jo + k-1 

k j l fc- 1 


(1-/3) 


n+1 — fc 


(1-/3) 


n+1 


+ E 


j - jo + k \ _ I j-jo+k-1 

k j I k -1 


thus proving the claim. 


(1 

□ 





















